1 link tagged with all of: benchmarking + s3 performance + trino + spark
Click any tag below to further narrow down your results
Links
A developer tested whether DuckDB could compete with Spark and Trino at large scales and found that a single powerful machine running DuckDB outperforms distributed clusters on 10TB queries—faster than Trino and half the cost of multi-node setups. The key insight: most time is spent waiting on S3 reads, not computation, so distribution overhead doesn't help.
- At 10TB scale, single-node DuckDB on an r7gd.4xlarge beats Trino on identical queries by 1.7-2.1x, and costs half as much as 5-13 node clusters despite being 2-3x slower than them
- S3 read rate, not compute speed, determines performance at scale—DuckDB's default single-threaded S3 reads leave CPUs 90% idle, but running 4x more threads than cores achieves 3x faster read rates
- DuckDB v2.0 (alpha) expects 2-3x faster S3 reads, potentially making single-node DuckDB the cheapest option for datasets up to 10TB